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ABSTRACT 

An  evaluation  is  made  of  the  NASS  multiple  frame  (MF)  survey  estimator  and  a  simple 
ratio  estimator  for  various  agricultural  items  using  operational  estimates  from  various  quar¬ 
terly  agricultural  surveys  between  June  1992  and  May  1997.  The  relative  mean  deviation 
(R-MD)  from  the  official  statistics  and  the  relative  root  mean  square  deviation  (R-RMSD) 
from  the  official  statistics  were  used  as  the  performance  criteria  for  each  estimator.  More 
specifically,  the  effect  of  the  list  frame  coverage  (which  at  the  time  of  this  analysis  did  not 
incorporate  agricultural  census  records)  on  the  R-RMSD  and  R-MD  of  these  estimators  is 
investigated.  The  empirical  evaluations  show  that  the  R-RMSD  tends  to  increase  for  both 
estimators  when  the  list  coverage  is  below  eighty  percent.  The  performance  of  the  MF  es¬ 
timator  is  similar  for  the  base  and  non-base  survey  periods  for  most  items,  and  the  ratio 
estimator  performs  similarly  to  the  MF  estimator  when  the  list  coverage  exceeds  eighty-five 
percent. 
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SUMMARY 

This  report  summarizes  an  evaluation  made  of  the  performance  of  NASS  multiple  frame 
(MF)  survey  estimators  and  that  of  a  ratio  estimator  for  various  agricultural  items  as  a 
function  of  the  level  of  list  frame  coverage  using  surveys  made  prior  to  the  inclusion  of 
agricultural  census  records.  The  list  frame  for  these  surveys  is  incomplete,  requiring  that 
additional  information  be  collected  using  an  area  frame.  The  level  of  coverage  of  the  list 
frame  can  vary  substantially  at  the  state  and  regional  level  for  some  agricultural  items.  Part 
of  the  motivation  for  looking  at  the  performance  as  a  function  of  list  coverage  is  to  determine 
whether  or  not  the  performance  of  the  ratio  estimator  would  be  comparable  to  that  of  the 
MF  survey  estimator  at  some  identifiable  level  of  list  coverage. 

The  quarterly  estimates  and  official  statistics  were  obtained  for  all  quarters  available  for 
various  labor  items  from  June,  1992  to  April,  1996.  and  for  various  crop  and  hog  items  from 
June,  1993  to  December,  1997.  No  algorithm  or  procedure  was  used  to  remove  outliers  (if 
any  existed)  in  the  survey  data  since  these  evaluations  were  made  using  state  level  summary 
data  from  the  Survey  Processing  System  and  the  record  level  data  were  not  available  for 
detecting  any  outliers.  The  items  considered  in  this  study  include: 


Labor  Items:  Total  Number  of  Hired,  Self-Employed  and  Unpaid  Workers 

Crop  Items:  Planted  Acreage  for  Corn,  Soybean,  Sorghum,  and  Winter  Wheat  as 
well  as  Harvested  Acreage  for  Hay  and  Winter  Wheat 

Hog  Items:  Total  Number  of  Hogs  and  Number  of  Sows  Farrowed 


For  each  item,  the  performance  of  the  MF  estimator  and  the  ratio  estimator  was  de¬ 
termined  by  comparing  the  estimates  to  the  official  statistics.  The  relative  mean  deviation 
(R-MD)  from  the  official  statistics  and  the  relative  root  mean  square  deviation  (R-RMSD) 
from  the  official  statistics  were  used  as  performance  criteria.  These  computations  were  made 
at  the  regional  level  for  each  agricultural  item. 

The  performance  of  the  MF  survey  estimator  varies  with  respect  to  list  coverage  for 
different  items.  The  R-RMSD  of  these  estimators,  including  the  ratio  estimator,  decreases 
with  an  increase  in  list  coverage  for  the  labor  items.  This  happens  only  in  those  cases  for  crop 
and  hog  items  that  do  not  have  a  good  list  frame  coverage.  The  estimates  most  affected  by 
incompleteness  of  the  list  frame  had  sizable  contributions  to  the  total  estimate  from  regions 
with  list  coverage  below  80%. 

The  performance  for  the  MF  survey  estimators  is  evaluated  for  the  base  and  non-base 
survey  periods  and  is  found  to  be  comparable.  However,  for  some  items  the  R-RMSD  in 
the  base  period  is  higher  than  in  the  non-base  period.  Also,  a  comparison  is  made  between 
the  performance  of  the  MF  survey  estimator  and  that  of  the  ratio  estimator  in  the  non¬ 
base  survey  periods.  Their  performances  are  similar  when  the  list  coverage  exceeds  85%; 
otherwise,  the  MF  survey  estimator  has  smaller  R-RMSD  than  the  ratio  estimator. 


NOTES  AND  REMARKS 


1.  The  present  study  is  limited  in  scope  since  it  focuses  on  the  investigation  of  the  dif¬ 
ference  between  NASS  MF  estimates  and  the  official  statistics  as  a  function  of  list 
coverage  for  each  of  the  agricultural  items  studied.  It  does  not  investigate  the  specific 
reasons  or  causes  for  the  difference  found. 

2.  No  formal  statistical  tests  are  conducted  to  assess  the  difference  observed  due  to  cor¬ 
relation  expected  between  regional  quarterly  estimates  used  in  the  study  which  would 
violate  the  basic  assumptions  of  a  standard  t-test.  Instead,  a  R-RMSD  or  R-MD  value 
is  considered  significant  if  its  magnitude  is  high,  particularly  50  percent  or  more  higher 
relative  to  those  for  other  items  or  estimators. 

3.  Base  period  refers  to  the  first  period  of  a  survey  cycle  in  the  year  that  the  survey 
estimate  is  made.  So  it  may  not  be  an  optimum  case  in  terms  of  NOL  sample  data. 
The  reason  for  this  was  to  have  some  consistency  across  items.  For  labor  items,  the 
DE  estimates  are  available  in  January,  April,  July  and  October;  the  base  period  is 
July.  For  hog  items,  the  ADMW  and  RWMW  estimates  are  available  in  March  and 
December;  the  base  period  is  December.  For  most  crop  items,  the  IMMW  estimates 
are  available  in  March  and  December.  For  Winter  Wheat,  the  IMMW  estimates  are 
available  for  March  and  September.  For  purposes  of  this  study,  March  is  used  as  the 
the  base  period  for  crop  items. 

4.  Evaluations  are  made  by  comparing  the  MF  estimates  and  the  ratio  estimates  with  the 
official  statistics  at  the  regional  level.  The  regions  are  different  for  different  agricultural 
items.  The  present  study  utilizes  the  regions  as  currently  used  by  NASS. 
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DEFINITIONS 


ADMW  Estimator  -  Multiple  frame  estimator  used  for  estimating  hog  items  by 
combining  a  revised  Adjusted  list  estimator  with  a  Modified  Weight  estimator  for  the 
NOL. 

DE  Estimator  -  Direct  Expansion  Estimator. 

IMMW  Estimator  -  Multiple  frame  estimator  used  for  estimating  crop  items  by 
combining  a  IMputed  weight  estimator  for  the  list  with  a  Modified  Weight  estimator 
for  the  NOL. 

MF  Estimator  -  A  Multiple  Frame  Estimator  combines  separate,  independently  com¬ 
puted  estimates  from  a  list  frame  and  the  NOL  component  of  an  area  frame  into  an 
estimate  of  the  total.  The  use  of  the  expression  “MF  Estimators”  in  this  report  refers 
to  the  DE  estimator  for  labor  items,  the  IMMW  estimator  for  crop  items,  and  the 
ADMW  and  RWMW  estimators  for  hog  items. 

MD  -  The  Mean  Difference  between  results  of  a  NASS  estimator  and  the  official 
statistics. 

NASS  -  National  Agricultural  Statistical  Service 

Official  Statistics  -  Agricultural  Statistical  Board  Final  Estimates 

RMSD  -  The  Root  Mean  Squared  Deviation  from  the  results  of  a  NASS  estimator  and 
the  official  statistics. 

R-MD  -  The  Relative  Mean  Difference  (see  Formula  1)  between  the  results  of  a  NASS 
estimator  and  the  official  statistics. 

R-RMSD  -  The  Relative  Root  Mean  Squared  Deviation  (see  Formula  2)  of  the  results 
of  a  NASS  estimator  and  the  official  statistics. 

RWMW  Estimator  -  Multiple  frame  estimator  used  for  estimating  hog  items  by 
combining  a  ReWeighted  estimator  for  the  list  with  a  Modified  Weight  estimator  for 
the  NOL. 
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INTRODUCTION 

The  National  Agricultural  Statistics  Service 
(NASS)  has  been  investigating  estimation 
approaches  that  would  minimize  (or  com¬ 
pletely  eliminate)  the  use  of  area-frame  sam¬ 
ples  in  adjusting  a  survey  estimate  for  in¬ 
completeness  of  the  list  frame.  These  inves¬ 
tigations  were  initiated  because  of  the  rela¬ 
tively  high  survey  cost  and  respondent  bur¬ 
den  associated  with  the  area  frame  samples 
and  also  because  of  the  poor  precision  of  the 
resulting  estimates  for  the  area  that  is  non¬ 
overlapping  with  the  list  frame  (NOL). 

Alternative  approaches  have  been  pro¬ 
posed  and  investigated  previously  as  dis¬ 
cussed  in  Chhikara.  et  al  (1995),  Rumburg, 
et  al  (1993)  and  Vogel  (1990).  Several  alter¬ 
native  estimators  were  considered  and  com¬ 
pared  to  the  currently  employed  direct  ex¬ 
pansion  (DE)  estimator  for  various  labor 
items  as  detailed  in  Perry,  et  al  (1997  and 
its  supplement  in  1998)  and  Spears,  et  al 
(1996,  1997).  Overall,  none  of  the  alter¬ 
native  estimators  consistently  matched  the 
performance  of  the  DE.  However,  in  some 
cases,  alternative  estimators  performed  com¬ 
parably  to  the  DE.  The  ratio  estimator  per¬ 
formed  well  in  some  cases  and  showed  the 
most  potential  among  the  alternative  esti¬ 
mators  that  rely  less  on  the  area  frame  than 
the  DE.  The  performance  to  a  large  extent 
depended  upon  the  size  of  the  list  frame  cov¬ 
erage. 

The  present  evaluation  study  is  extended 
to  crop  and  hog  items  for  which  NASS  quar¬ 
terly  survey  estimates  and  official  statistics 
were  available  from  June  of  1993  to  Decem¬ 
ber  of  1997.  For  the  sake  of  comparison 
across  various  agricultural  items,  labor  items 
are  also  included  in  the  study.  The  perfor¬ 
mance  of  each  multiple  frame  (MF)  survey 
estimator  is  examined  for  both  the  base  and 


non-base  survey  periods  as  well  as  the  com¬ 
bined  (all  survey)  periods.  The  ratio  estima¬ 
tor  is  considered  only  for  the  non-base  sur¬ 
vey  period,  and  its  performance  is  evaluated 
for  that  period.  The  performance  of  a  sur¬ 
vey  estimator  is  based  on  its  value  relative 
to  the  corresponding  official  statistics  value. 
No  algorithm  or  procedure  could  be  used  to 
remove  outliers  (if  any  existed)  in  the  survey 
data  due  to  the  nature  of  the  data  utilized 
in  this  study. 

The  ratio  estimate  in  a  non-base  survey 
period  is  obtained  by  multiplying  the  list  es¬ 
timate  for  that  period  by  the  ratio  of  total 
estimate  to  the  list  estimate  as  determined  in 
the  base  survey  period.  The  performance  of 
the  ratio  estimator  is  examined  as  a  function 
of  the  list  frame  coverage  and  compared  to 
that  of  the  MF  survey  estimator  for  various 
agricultural  items. 

NASS  SURVEYS  AND  ESTI¬ 
MATORS 

The  survey  frequency  and  estimation 
methodology  for  NASS  sample  surveys 
varies  for  different  agricultural  items.  The 
survey  estimation  is  based  on  a  multiple 
frame  approach  that  combines  separate, 
independently  computed  estimates  from  a 
list  frame  and  the  NOL  component  of  an 
area  frame  into  an  estimate  of  the  total. 
NASS  currently  utilizes  direct  expansion 
estimators  for  both  the  list  and  NOL 
components  as  described  in  Kott  (1990). 

The  following  items  were  included  in  the 
evaluation  of  NASS  MF  estimators. 


Labor  Items 

Quarterly  labor  surveys  are  conducted  in 
January,  April,  July  and  October  to  esti¬ 
mate  various  characteristics  for  hired,  self- 
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employed  and  unpaid  workers.  The  items 
considered  in  this  study  include  the  total 
number  of  hired,  self-employed  and  unpaid 
workers.  July  is  used  as  the  base  survey  pe¬ 
riod  and  the  following  October,  January,  and 
April  are  the  non-base  periods  used  to  deter¬ 
mine  the  ratio  estimates.  Each  item’s  quar¬ 
terly  estimates  are  computed  separately  for 
the  17  farm  labor  regions  and  these  are  com¬ 
pared  with  the  corresponding  official  statis¬ 
tics  for  their  evaluation. 


Crop  Items 

Quarterly  crop  and  stock  surveys  are  con¬ 
ducted  to  estimate  acreage,  yield  and  pro¬ 
duction  for  crop  and  stock  items.  In  this 
study,  only  the  crop  acreage  estimates  are 
evaluated.  To  estimate  crop  acreage,  NASS 
utilizes  direct  expansion  estimates  computed 
using  the  IMMW  estimator  which  combines 
an  imputed  weight  estimator  for  the  list  with 
a  modified  weight  estimator  for  the  NOL. 
The  crop  items  considered  in  this  study 
include  (1)  the  planted  acreage  for  corn, 
sorghum,  soybean  and  winter  wheat,  and 
(2)  the  harvested  acreage  for  hay  and  win¬ 
ter  wheat.  The  IMMW  estimates  are  avail¬ 
able  in  March  and  December  for  the  items 
other  than  for  winter  wheat  planted  and  har¬ 
vested,  for  which  estimates  are  available  in 
March  and  September.  So  in  this  study, 
March  is  the  base  survey  period  for  all  crop 
items  and  the  following  December  is  the  non¬ 
base  period  for  the  ratio  estimator  of  corn, 
sorghum,  soybeans,  and  hay;  and  September 
is  the  non-base  period  for  the  ratio  estima¬ 
tor  of  winter  wheat  planted  and  harvested. 
The  crop  regions  are  those  currently  used  by 
NASS. 


Hog  Items 

Quarterly  hog  surveys  are  conducted  to  esti¬ 
mate  various  hog  items.  NASS  direct  expan¬ 


sion  estimates  of  hog  items  use  the  ADMW 
estimator  which  combines  a  revised  adjusted 
list  estimator  with  a  modified  weight  estima¬ 
tor  for  the  NOL.  The  RWMW  estimator  is 
also  used  for  the  hog  items  which  employs 
a  reweighted  estimator  for  the  list.  The  hog 
items  considered  in  this  study  include  the  to¬ 
tal  number  of  hogs  and  the  number  of  sows 
farrowed.  The  ADMW  and  RWMW  esti¬ 
mates  are  available  for  these  items  in  March 
and  December.  The  estimates  of  the  num¬ 
ber  of  marketed  hogs  in  various  weight  cate¬ 
gories  were  also  evaluated.  December  is  the 
base  period  and  the  following  March  is  the 
non-base  period  for  the  ratio  estimator  of 
hog  items.  The  quarterly  estimates  are  com¬ 
puted  at  the  regional  level;  the  hog  regions 
are  the  same  as  those  used  by  NASS. 

Due  to  changes  in  the  Hog  Survey  pro¬ 
gram,  the  March  1997  survey  included  only  a 
list  frame  sample.  The  NOL  component  was 
modeled  based  on  post-stratified  December 

1996  MF  to  list  frame  ratios.  The  March 

1997  MF  estimate  was  treated  the  same  as 
the  ADMW  estimates  from  previous  years  in 
the  R-RMSD  and  R-MD  calculations 


Ratio  Estimator 

A  ratio  estimate  for  a  non-base  survey  period 
can  be  obtained  using  a  multiple  frame  es¬ 
timate  from  a  base  survey  period.  Both  the 
list  frame  and  the  NOL  must  be  estimated 
during  a  base  survey  period.  For  non-base 
survey  periods,  the  list  frame  estimate  is 
multiplied  by  the  ratio  of  the  multiple  frame 
estimate  (list  +  NOL)  to  the  list  frame  es¬ 
timate  for  the  base  survey  period  to  obtain 
the  ratio  estimate.  (A  ratio  estimate  can  be 
different  from  a  multiple  frame  estimate  only 
in  a  non-base  period.)  Hence  it  is  evaluated 
only  for  the  non-base  periods. 
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Table  1:  Summary  of  Regional  List  Coverage  (in  percent)  for  Crops 


Corn 

Soybean 

Planted 

Sorghum 

Winter  Wheat 

|  Hay 

Harvested 

Winter  Wheat 

Min 

79.4 

77.8 

76.4 

78.4 

30.6 

78.4 

Mean 

87.6 

86.1 

87.5 

87.3 

70.7 

88.0 

Max 

93.2 

94.6 

96.6 

95.0 

92.9 

95.0 

Table  2:  Summary  of  Regional  List  Coverage  (in  percent)  for  Hogs  and  Workers 


Total 

Hogs 

Sows  Farrowed 

Hired 

Workers 

Self-Employed 

Unpaid 

Min 

69.8 

59.0 

72.6 

38.1 

38.7 

Mean 

90.2 

89.9 

81.7 

52.0 

61.0 

Max 

99.3 

99.6 

91.8 

76.2 

84.1 

EMPIRICAL  EVALUATIONS 

The  performance  of  the  estimators  described 
in  the  previous  section  is  evaluated  using  op¬ 
erational  estimates  from  the  quarterly  agri¬ 
cultural  surveys  from  1992-1997  for  regions 
in  the  United  States.  For  labor  items,  the 
study  period  is  from  June  1992  to  April 
1996,  whereas  for  crop  and  hog  items,  it  is 
from  June  1993  to  December  1997.  The  re¬ 
gional  level  estimates  are  compared  to  the 
official  statistics  which  represent  the  final  re¬ 
vised  NASS  Board  estimates,  and  their  per¬ 
formance  is  examined  as  a  function  of  list 
frame  coverage. 

The  list  frame  coverage  is  estimated  by 
the  ratio  of  the  average  list  frame  estimate 
to  the  average  multiple  frame  estimate  in  the 
base  survey  period,  obtained  for  the  entire 
study  period.  The  level  of  coverage  of  the 
list  frame  varies  substantially  at  the  state 
and  regional  level  for  some  of  the  agricul¬ 
tural  items.  The  list  coverage  for  items  other 
than  self-employed  workers,  unpaid  workers, 
hay  harvested  and  sows  farrowed  is  mostly 


between  70%  and  99%  at  the  regional  level. 
It  ranges  between  31%  and  93%  for  har¬ 
vested  hays,  between  38%  and  76%  for  self- 
employed  workers,  between  39%  and  84%  for 
unpaid  workers,  and  between  59%  and  100% 
for  sows  farrowed.  The  range  and  mean  per¬ 
cent  list  coverage  for  regions  are  summarized 
in  Tables  1  and  2. 

The  performance  criteria  are  the  relative 
root  mean  squared  deviation  (R-RMSD)  and 
the  relative  mean  deviation  (R-MD),  both 
measured  from  the  official  statistics  (OS). 
Letting  Yt  and  Yos,i  represent  the  item  esti¬ 
mates  for  the  i th  survey  period  correspond¬ 
ing  to  the  MF  estimator  and  the  official 
statistics,  respectively,  the  R-RMSD  is  given 
by 


R-RMSD(f)  =  VnEt~1r|^  ,,  (1) 

n  Ej=i  Yos,i 

and  the  relative  mean  deviation  (R-MD)  is 
given  by 
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Table  3:  Average  R-MD 


Item  Estimated 

NASS  MF  Estimate 

All  Base  Non-base 

Ratio 

Labor 

Hired  Workers 

0.031 

0.056 

0.019 

0.046 

Self-Employed  Workers 

0.018 

0.039 

0.010 

0.042 

Unpaid  Workers* 

0.034 

0.064 

0.017 

0.086 

Crops 

Corn  Planted 

0.016 

0.021 

0.013 

0.020 

Soybean  Planted 

0.006 

-0.028 

0.019 

0.022 

Sorghum  Planted 

-0.063 

-0.082 

-0.057 

-0.028 

Winter  Wheat  Planted 

-0.066 

-0.062 

-0.068 

-0.064 

Hay  Harvested 

0.008 

-0.004 

0.011 

0.006 

Winter  Wheat  Harvested 

-0.025 

0.013 

-0.042 

-0.042 

Hogs 

Total  Hogs 

-0.057 

-0.056 

-0.057 

-0.058 

Sows  Farrowed 

-0.029 

-0.042 

-0.026 

-0.026 

7  Results  from  the  Mountain  III  Region  were  not  included  because  of  large  discrepancies  between  the  direct 
expansion  estimates  and  the  official  statistics,  possibly  due  to  some  outlier(s)  in  the  record  level  data. 


R-MD(y)  =  n  .  }°s^\  (2) 

v  '  1  v  v  ' 

n  ^i=l  rOS,i 

where  n  denotes  the  number  of  survey  peri¬ 
ods  for  which  estimates  are  computed.  For 
the  MF  estimators  these  quantities  are  com¬ 
puted  at  the  regional  level  separately  for  the 
base  and  non-base  survey  periods  as  well  as 
for  all  periods  combined.  Since  the  ratio 
estimates  are  only  computed  for  the  non¬ 
base  periods,  its  R-RMSD  and  R-MD  are 
obtained  only  for  the  non-base  periods. 

For  each  agricultural  item,  the  average 
R-MD  and  average  R-RMSD  are  computed 
across  all  regions  by  weighting  the  regions  in 


terms  of  their  list  coverage.  These  results  are 
summarized  in  Tables  3  and  4,  respectively. 


Appendix  A  contains  (in  Figures  Al- 
All)  bubble  plots  of  the  R-RMSD  for  each 
region  versus  the  list  coverage  for  that  re¬ 
gion  for  each  of  the  items  evaluated  in  this 
study.  The  size  of  each  bubble  is  propor¬ 
tional  to  the  size  of  the  contribution  of  the 
estimate  from  that  region  to  the  national  es¬ 
timate.  For  each  item,  the  R-RMSD  is  plot¬ 
ted  for  the  MF  estimator  using  (1)  all  sur¬ 
vey  periods,  (2)  base  survey  periods,  and  (3) 
the  non-base  survey  periods;  and  for  the  ra¬ 
tio  estimator  using  non-base  survey  periods. 
Similar  plots  for  the  R-MD  are  included  in 
Appendix  B. 
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Table  4:  Average  R-RMSD 


Item  Estimated 

NASS  MF  Estimate 
All  Base  Non-base 

Ratio 

Labor 

Hired  Workers 

0.084 

0.084 

0.079 

0.118 

Self-Employed  Workers 

0.054 

0.065 

0.0463 

0.100 

Unpaid  Workers* 

0.098 

0.103 

0.075 

0.212 

Crops 

Corn  Planted 

0.054 

0.057 

0.050 

0.057 

Soybean  Planted 

0.061 

0.061 

0.058 

0.067 

Sorghum  Planted 

0.150 

0.183 

0.123 

0.117 

Winter  Wheat  Planted 

0.091 

0.087 

0.091 

0.088 

Hay  Harvested 

0.104 

0.117 

0.095 

0.116 

Winter  Wheat  Harvested 

0.090 

0.104 

0.076 

0.076 

Hogs 

Total  Hogs 

0.059 

0.065 

0.055 

0.065 

Sows  Farrowed 

0.070 

0.064 

0.071 

0.079 

t 


Results  from  the  Mountain  III  Region  were  not  included  because  of  large  discrepancies  between  the  direct 


expansion  estimates  and  the  official  statistics,  possibly  due  to  some  outlier(s)  in  the  record  level  data. 


Labor  Items 

The  average  R-MD  is  positive  for  all  labor 
items  at  the  regional  level.  The  R-MD  is 
higher  on  average  for  the  ratio  estimator 
than  for  the  DE  estimator  for  the  non-base 
period.  However,  in  all  cases,  the  bias  re¬ 
flected  by  the  positive  R-MD  is  small  and 
statistically  insignificant. 

Figures  A1-A3  show  that  the  R-RMSD 
decreases  as  list  coverage  increases  for  the 
DE  and  ratio  estimators.  The  decrease  is 
substantially  more  for  the  ratio  estimator. 

Since  the  R-RMSD  for  the  non-base  pe¬ 
riod  is  smaller  for  the  DE  than  for  the  ratio 
estimator,  the  DE  has  better  performance 
than  the  ratio  estimator. 


Crop  Items 

For  planted  acreage,  the  R-MD  is  on  aver¬ 
age  negative  for  the  IMMW  estimator  for 
sorghum,  soybean  (base  period)  and  winter 
wheat.  However,  these  are  statistically  in¬ 
significant  for  every  item.  Figures  in  Ap¬ 
pendix  A  for  the  crops  show  that  the  R- 
RMSD  decreases  as  list  coverage  increases 
for  corn  and  soybean,  but  fairly  random  in 
other  cases.  The  average  R-RMSD  values 
given  in  Table  4  show  that  the  performance 
is  similar  in  the  base  and  non-base  periods 
for  the  IMMW  direct  expansion  estimator. 
Also,  the  direct  expansion  and  ratio  estima¬ 
tor  have  comparable  performance. 

For  harvested  acreage,  the  R-MD  is  on 
average  negative,  yet  statistically  insignifi¬ 
cant,  for  the  IMMW  estimator  for  winter 


5 


Figure  1:  Ratio  of  Mean  R-RMSD  versus  Mean  List  Coverage 

Ratio  Versus  NASS  Estimator  (Non-Base  Period) 
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wheat.  The  average  R-MD  for  hay  is  al¬ 
most  equal  to  zero.  There  is  no  decrease  in 
R-RMSD  as  list  coverage  increases  for  har¬ 
vested  acreage  of  any  of  the  items.  The  per¬ 
formances  of  the  IMMW  direct  expansion 
and  ratio  estimators  are  again  comparable. 


Hog  Items 

The  R-MD  is  on  average  negative  for  the 
ADMW  estimator,  although  statistically  in¬ 
significant,  and  R-RMSD  decreases  slightly 
as  list  coverage  increases  for  the  total  hogs 
and  the  number  of  sows  farrowed.  Similar  re¬ 
sults  are  obtained,  though  not  included  here, 
for  the  RWMW  estimator.  The  performance 
of  the  ADMW  (or  RWMW)  is  similar  in  the 
base  and  non-base  periods,  and  the  ratio  es¬ 
timator  performs  similarly,  as  well.  These 


conclusions  follow  from  Tables  3  and  4,  and 
Figures  A4  and  A5  given  in  Appendix  A. 

Evaluations  of  the  MF  survey  estimates 
for  market  hogs  in  various  weight  categories 
were  also  made.  The  performance  of  the 
individual  weight  category  estimates  was 
found  to  be  somewhat  inconsistent,  so  it  is 
not  reported  here. 

CONCLUSIONS 

A  number  of  conclusions  follow  as  a  result 
of  this  study  of  the  performance  of  the  MF 
survey  and  ratio  estimators  for  various  agri¬ 
cultural  items. 

•  The  ratio  estimator  has  R-RMSD 
within  20%  of  that  of  the  MF  estima¬ 
tors  for  all  items  other  than  the  la- 
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Figure  2:  Ratio  of  Mean  R-RMSD  versus  Mean  List  Coverage 

NASS  Estimator  (Base  versus  Non-Base  Periods) 
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bor  items.  When  the  list  coverage  ex¬ 
ceeds  85%,  the  ratio  estimator  has  per¬ 
formance  comparable  to  the  MF  esti¬ 
mators.  These  results  can  be  seen  in 
Figure  1.  It  shows  the  ratio  of  the 
mean  R-RMSD  for  the  ratio  estima¬ 
tor  to  that  of  the  MF  survey  estimator 
as  a  function  of  the  mean  list  cover¬ 
age  for  each  of  the  items  included  in 
this  study.  The  mean  is  obtained  as  a 
weighted  average  across  all  regions  in 
each  case. 

Remark:  The  abbreviations  for  the 
items  used  in  the  legend  are:  “Hired” 
for  number  of  hired  workers,  “Self” 
for  number  of  self-employed  workers, 
“Unpd”  for  number  of  unpaid  work¬ 
ers,  “CrnPl”  for  corn  planted  acreage, 
“HayHv”  for  hay  harvested  acreage, 


“SoyPl”  for  soybean  planted  acreage. 
“SrgPl”  for  sorghum  planted  acreage, 
“WWhHV”  for  winter  wheat  har¬ 
vested  acreage,  “WWhPl”  for  win¬ 
ter  wheat  planted  acreage,  “Sows”  for 
number  of  sows  farrowed  and  “Hogs” 
for  total  number  of  hogs. 

•  The  MF  estimators  are  comparable 
(within  20  percent)  in  their  R-RMSD 
for  base  and  non-base  periods  in  all 
cases  except  self-employed  and  unpaid 
workers,  sorghum  planted  acreage  and 
harvested  acreage  for  hay  and  winter 
wheat.  In  these  cases,  the  R-RMSD  in 
the  base  period  is  higher  by  20  to  50 
percent.  See  Figure  2  which  plots  the 
ratio  of  the  mean  R-RMSD  in  the  base 
period  to  that  of  the  non-base  period 
in  a  manner  similar  to  Figure  1. 


< 


•  The  R-RMSD  of  the  MF  estimator  de¬ 
creases  with  increased  list  coverage  for 
all  labor  items  and  soybean  planted 
acreage.  In  most  other  cases,  a  de¬ 
crease  in  R-RMSD  as  a  function  of  list 
coverage  is  seen  with  respect  to  regions 
that  are  substantially  contributing  to 
the  total  estimate  for  an  item. 

•  The  average  R-MD  of  the  ratio  esti¬ 
mator  is  comparable  to  that  of  the  MF 
estimator  for  all  items  other  than  the 
labor  items. 

•  The  average  R-MD  of  the  MF  estima¬ 
tors  are  comparable  for  base  and  non¬ 
base  periods  in  all  cases  except  labor 
items.  It  is  higher  by  a  multiple  of  al¬ 
most  3  or  more  in  the  base  period  for 
the  labor  items. 

•  The  R-MD  is  negative  for  most  regions 
of  planted  acreage  of  sorgum  and  win¬ 
ter  wheat,  and  it  is  positive  for  all  la¬ 
bor  items. 


Overall,  the  performance  of  the  MF  sur¬ 
vey  estimator  varies  with  respect  to  the  list 
coverage  for  different  items.  The  estimates 
most  affected  by  incompleteness  of  the  list 
frame  have  sizeable  contributions  from  re¬ 
gions  with  list  coverage  below  80%  which  can 
be  seen  from  the  Figures  in  Appendix  A; 
for  example,  in  the  cases  of  hay  harvested 
acreage  and  unpaid  workers,  major  contri¬ 
butions  to  their  estimates  come  from  regions 
having  list  coverage  of  less  than  80%. 

The  ratio  and  MF  survey  estimators  have 
similar  performance  in  cases  where  the  list 
coverage  exceeds  85%. 


8 


REFERENCES 

Allen,  Rich  (1997).  “Statistical  Defensibility 
as  Used  by  U.S.  Department  of  Agricul¬ 
ture,  National  Agricultural  Statistics  Ser¬ 
vice.”  Journal  of  Official  Statistics ,  8:4, 
481-498. 

Chhikara,  Raj  S.,  Perry,  Charles  R.,  Deng, 
Lih-Yuan,  Iwig  William  C.,  Spears  Floyd 
M.  and  Cowles,  Susan  (1995).  “Post- 
Stratification  and  Efficient  Estimation  in 
U.S.  Agricultural  Labor  Surveys”.  ASA 
Proceedings  of  Survey  Research  Methods, 
1995. 

Kott,  P.S.  (1990).  “Mathematical  Formulae 
for  the  1989  Survey  Processing  System 
(SPS)  Summary”.  USDA-NASS  SRB 
Research  Report  No.  SRB-90-08,  May, 
1990. 

Perry,  Charles  R..  Chhikara,  Raj  S.,  Spears, 
Floyd  M.,  and  Cowles,  Susan  (1997).  “An 
Evaluation  of  List-Only,  Reweighted,  and 
Other  Estimators  for  U.S.  Agricultural 
Labor  Surveys”  USDA-NASS  SRB  Re¬ 
search  Report  No.  RD-97-06,  October, 
1997. 

Perry,  Charles  R.,  Chhikara,  Raj  S.,  Spears, 
Floyd  M.  (1998).  “Comparison  of  Labor 
Survey  Estimates  to  the  Board.”  Supple¬ 
ment  to  USDA-NASS  SRB  Research  Re¬ 
port  No.  RD-97-06,  January,  1998. 

Perry,  Charles  R.,  Chhikara,  Raj  S.,  Deng, 
Lih-Yuan,  Iwig,  William  C.,  and  Rum- 
burg,  Scot  (1993).  “Generalized  Post¬ 
stratification  Estimators  in  the  Agricul¬ 
tural  Labor  Survey”,  SRB  Research  Re¬ 
port  No.  SRB-93-04,  Washington,  D.C., 
July,  1993. 

Rumburg,  Scot,  Perry,  Charles  R.,  Chhikara, 
Raj  S.,  and  Iwig,  William  C.  (1993). 


Analysis  of  a  Generalized  Post-Stratifica¬ 
tion  Approach  for  the  Agricultural  Labor 
Survey.  SRB  Research  Report  No.  SRB- 
93-05,  July,  1993. 

Spears,  Floyd,  Chhikara,  Raj  S.,  and  Perry, 
Charles  R.  (1998).  “An  Investigation 
of  Incompleteness  of  List  Frames  in  US 
Agricultural  Surveys”.  ASA  Proceedings 
of  Survey  Research  Methods,  1998. 

Spears,  Floyd  M.,  Chhikara,  Raj  S.  . 
Perry,  Charles  R.,  and  Cowles,  Susan 
(1997).  “An  Evaluation  of  Alternative 
USD  A  Agricultural  Labor  Survey  Esti¬ 
mators”.  ASA  Proceedings  of  Survey  Re¬ 
search  Methods,  1997. 

Spears,  Floyd  M.,  Chhikara,  Raj  S.,  Perry. 
Charles  R.,  Iwig,  William  C.  and  Cowles, 
Susan  (1996).  “Agricultural  Labor  Es¬ 
timation  Using  Only  List-Frame  Sam¬ 
pling”.  ASA  Proceedings  of  Survey  Re¬ 
search  Methods,  1996. 

Vogel,  Frederic  A.  (1995)  “The  Evolution 
and  Development  of  Agricultural  Statis¬ 
tics  at  the  United  States  Department  of 
Agriculture.”  Journal  of  Official  Statis¬ 
tics. ,  11:2,  161-180. 

Vogel,  Frederic  A.  (1990).  “Straw- 
man  Proposal  for  Multiple  Frame  Sam¬ 
pling”  NASS  Internal  Memos,  October- 
November,  1990. 


9 


APPENDIX  A:  REGIONAL  LEVEL  R-RMSD  RESULTS 

List  of  Figures 

A1  Hired  Workers  .  11 

A2  Self-Employed  Workers  .  12 

A3  Unpaid  Workers .  13 

A4  Total  Hogs  .  14 

A5  Sows  Farrowed  .  15 

A6  Corn  Planted  Acreage  .  16 

A7  Hav  Harvested  Acreage .  17 

A8  Soybean  Planted  Acreage  .  18 

A9  Sorghum  Planted  Acreage  .  19 

A 10  Winter  Wheat  Planted  Acreage  .  20 

All  Winter  Wheat  Harvested  Acreage  .  21 

Using  the  Quarterly  Agricultural  Surveys  from  states  or  agricultural  regions  in  the  United 
States,  estimates  of  various  labor  items  from  1992-96  and  various  hog  and  crop  items  from 
1993-97  were  compared  to  the  official  statistics  at  the  regional  level  using  the  relative  root 
mean  squared  deviation  (R-RMSD)  as  the  performance  criterion. 
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Figure  A1:  R-RMSD  versus  List  Coverage 


Total  Hired  Workers  (Region) 
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Figure  A2:  R-RMSD  versus  List  Coverage 


Total  Self-Employed  Workers  (Region) 
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Figure  A3:  R  -  RMSD  versus  List  Coverage 

Total  Unpaid  Workers  (Region) 
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Figure  A4:  R-RMSD  versus  List  Coverage 
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Figure  A5:  R-RMSD  versus  List  Coverage 

Sows  Farrowed:  Mar -May  (Region) 
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Figure  A6:  R-RMSD  versus  List  Coverage 

Corn  -  All  Planted  Acreage  (Region) 


Imp  List  +  Mod  Wt  NOL  (All  Penods) 

Imp  List  +  Mod  Wt  NOL  (Base  Periods) 

[JO 

R-RMSD 

0.20 

5.15 

0.15 

0 

0 

0 

o  0 

3 13 

0 

0 

5.15 

0  °  °  ‘  0 

0  0° 

0 

5.55 

0  0  d 

0  o  0  n  0 

o  @(P 

0 

0 

5  50 

f!  Oil 

5. 

15  5.80  3.85  3.55  1.55  1.50 

List  Coverage 

5.15  3.85  5.85  5,55  5.55  1.55 

List  Coverage 

Imp  List  +  Mod  Wt  NOL  (NonBase  Periods) 


o 

0 

0 

o 

0  0 

o  Cp  °  0 

n  0 
u  0 


15  5.83  1.55  1.00 

List  Coverage 


Ratio  Estimator  (NonBase  Periods) 


R-RVSD 

0.20' 


5,10 


5.50 


o 


5.15  5.85 


o  o 


0 

o  o 


"■  I  r"1"''  . I  ,-r  r~'~'  '  1  r~ 1  I  1  i  1  1  1  '  i  ''  T 

5.85  8.55  5.55  1.55 

List  Coverage 


16 


Figure  A7:  R-RMSD  versus  List  Coverage 
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Figure  A8:  R-RMSD  versus  List  Coverage 
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Figure  A9:  R-RMSD  versus  List  Coverage 

Sorghum  -  All  Planted  Acreage  (Region) 


Imp  List  +  Mod  Wt  NOL  (All  Periods) 

H-F.MSD 

0.8 

0.1 

0.5 

0.5 

0 

0.4 

0.3 

0 

0 

0.2 

0  0  0  0  0  0 

0.1 

o 

) 

o 

o 

(1 IT 

0.15  0.80  0.85  0.31  0.55  1.00 

List  Coverage 

E-8S85 

Imp  List  +  Mod  Wl  NOL  (Base  Periods) 

0.3 

5.1 

0.5 

3.5 

0 

0.4 

0 

0.3 

0 

0 

3.2 

!)  1 

o 

5 

O 

o 

o 

o 

J.I 

0.0 

0 

c 

O 

o 

15  0.80  0.85  0.30  0.35  1.00' 

List  Coverage 

Imp  List  +  Mod  Wt  NOL  (NonBase  Periods) 


HUSO 

0.1 

o.i 

0.0 


0.3 

0.2 

0.1 


o  n  o  (p°  ° 

o°U  0 


0.15 


0.85  0.5!  0.55  1.00 

List  Ccverage 


j-jOlSC 

0.8 

0.1 

0.5 

0.5 

0.1 


0.2 


Ratio  Estimator  (NonBase  Periods) 


0 


0 


o 


CO 


0 


o 


0.30 


List  Coverage 


19 


Figure  A10:  R-RMSD  versus  List  Coverage 
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Figure  All:  R-RMSD  versus  List  Coverage 
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APPENDIX  B:  REGIONAL  LEVEL  R-MD  RESULTS 
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Using  the  Quarterly  Agricultural  Surveys  from  states  or  agricultural  regions  in  the  United 
States,  estimates  of  various  labor  items  from  1992-96  and  various  hog  and  crop  items  from 
1993-97  were  compared  to  the  official  statistics  at  regional  level  using  the  relative  mean 
deviation  (R-MD)  as  the  performance  criterion. 
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Figure  B1:  R-MD  versus  List  Coverage 
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Figure  B2:  R  -  MD  versus  List  Coverage 
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Figure  B3:  R-MD  versus  List  Coverage 
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Figure  B4:  R-MD  versus  List  Coverage 
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Figure  B5:  R-MD  versus  List  Coverage 
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Figure  B6:  R-MD  versus  List  Coverage 
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Figure  B7:  R-MD  versus  List  Coverage 
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Figure  B8:  R  -  MD  versus  List  Coverage 

Soybean  -  All  Planted  Acreage  (Region) 


Imp  List  +  Mod  Wt  NOL  (All  Periods) 


:.r 


o  o 
o 


-o- 


-Q- 


3.90  3.95  3.53  0.35 

List  Coverage 


Imp  List  +  Mod  Wt  NOL  (Base  Periods) 


List  Coverage 


30 


S.J5 


Figure  B9:  R-MD  versus  List  Coverage 

Sorghum  -  All  Planted  Acreage  (Region) 
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Figure  BIO:  R-MD  versus  List  Coverage 

Winter  Wheat  -  All  Planted  Acreage  (Region) 
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Figure  B11:  R-MD  versus  List  Coverage 
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